Skip to content

Require unique scratch filenames and define the structurally-red-anchor path - #121

Merged
dmccoystephenson merged 2 commits into
mainfrom
fix/scratch-collisions-and-unreachable-anchor
Aug 16, 2026
Merged

Require unique scratch filenames and define the structurally-red-anchor path#121
dmccoystephenson merged 2 commits into
mainfrom
fix/scratch-collisions-and-unreachable-anchor

Conversation

@dmccoystephenson

Copy link
Copy Markdown
Member

Summary

Three defects were observed empirically during a 10-way parallel dev-loop fan-out on 2026-08-15. All three are properties of the base template rather than of any one generated skill, so they are fixed here.

1. Shared scratchpad filenames collide between concurrent cycles — motivated by dmccoystephenson/simpleskills-dev-loop#9. A food-spoilage cycle silently overwrote the scratchpad file holding a SimpleSkills PR body mid-cycle. Nothing was mis-posted only because gh pr create --body-file had already run; a slightly different ordering (write body, wait on CI, then use the file) would have published one repository's PR description onto another repository's PR.

  • Phase 3 gains a universal rule requiring scratch filenames unique per repo and per cycle (<repo>-<branch>-pr-body.md, never a generic pr-body.md), requiring the file to be written immediately before the command that consumes it, and requiring re-verification via gh pr view <number> --json body where a body has to survive a wait.
  • Phase 4's gh pr create fence is switched from --body "..." to --body-file, which is what the neighbouring command-substitution rule already prescribes for every other long body.
  • Phase 4 step 5's example scratch path is made per-branch rather than a fixed .self-review-scratch.md.
  • The concurrent-session entry in Edge cases, which previously covered only the shared tree and the shared origin/<default-branch>, now names the scratchpad as a third contended resource and notes that unlike the first two it fails silently.

2. No disposition for a repository whose CI can never be green — motivated by dmccoystephenson/beyond-nations-dev-loop#1. On Preponderous-Software/beyond-nations the Unity jobs have failed on every recorded run since February 2026 because the license secrets are empty, so the green-anchor merge gate can never be satisfied and every cycle terminates in a hand-off regardless of quality — including documentation-only changes for which those jobs carry no signal either way.

  • Phase 1 gains a triage step establishing whether a viable merge path exists (have the required workflows ever succeeded on the base branch?) and recording the answer, the cause, and any tracking issue.
  • Phase 4 step 1 gains a companion paragraph to the existing "Green CI is not verification when CI's scope excludes the changed files": a per-job signal assessment, with three explicit conditions under which a job carries no signal for the diff. Such a job is not counted as blocking — and is not counted as verification either, falling under the existing UNVERIFIED scope gate.
  • Phase 8 gains a paragraph requiring the hand-off to name the blocking condition, the per-job assessment, and the tracking issue, rather than reporting an ordinary "awaiting review" — the two have entirely different remedies.
  • A matching Edge cases entry is added next to the existing "external anchor cannot run" entry.

3. A vacuously green build anchor — motivated by dmccoystephenson/food-spoilage-dev-loop#3. ./gradlew test prints BUILD SUCCESSFUL on :test NO-SOURCE, and the pytest/npm equivalents exit 0 on "no tests ran".

  • The Phase 3 verification block gains a rule requiring the executed-test count to be read before a PASS is recorded, and forbidding a zero-test green from being reported as "tests pass".
  • Phase 3's implementation summary now asks for the test count alongside the command that produced the verdict.

The second half of that issue (exporting JAVA_HOME before invoking ./gradlew) is a repo-specific finding and was deliberately not back-ported here, per CLAUDE.md's "What belongs here vs. in generated skills".

No new placeholders are introduced and no phase numbers change, so no substitution-table or README changes are required.

Research grounding

Defects 2 and 3 both implement RESEARCH.md §1 (self-critique without an external signal is unreliable). A required job that has never passed, and a green that executed zero tests, are both anchors that carry no information; treating either as the external signal degrades the self-review to exactly the unanchored critique that finding shows is neutral-to-harmful. The existing "Green CI is not verification when CI's scope excludes the changed files" rule from PR #54 is the same argument applied to a different way an anchor can be uninformative, and these two changes are written as its neighbours. An Implementations entry is added under §1.

Defect 1 (scratch-file collisions) is a harness-concurrency hazard, not a model-behavior question: no finding in RESEARCH.md applies, and none is claimed.

Doc sync check

  • README.md's "What it does" Step list still matches create-dev-loop.md's Steps 1:1 — no Steps were added, removed, or renamed
  • Every {{placeholder}} added or changed has a corresponding Step 4 substitution-table row — no new placeholders; the added text reuses {{EXTERNAL_SIGNAL_LABEL}}, {{DEFAULT_BRANCH}}, and {{TEST_CMD}}
  • RESEARCH.md updated (an Implementations entry under §1 for the anchor-integrity changes)

Test plan

  • python3 scripts/check_docs.py — passes (0 errors).
  • python3 -m pytest tests/test_check_docs.py -q — 13 passed.
  • The behavioral anchor (running /create-dev-loop against a real repository end-to-end) was not run for this change, and is marked UNVERIFIED. The change is additive prose inside the template block: fences remain escaped, phase numbers and the Step list are untouched, and no placeholder was added, so the mechanical checks cover the failure modes a generation run would surface. A reviewer who wants the stronger signal can regenerate any low-stakes skill from this branch.

Propagation

Propagation to existing generated skills is still required and is not done here. Roughly 69 <slug>-dev-loop instances exist, all predating this change, and per CLAUDE.md's "Promoting a rule into the template" the template only fixes drift forward. Every instance carrying a <!-- template-version: ... --> older than this commit needs a retrofit pass through the generic maintain-dev-loop skill — a separate wave, not this PR. The three motivating issues in the instance repos are referenced above as motivation only and are deliberately not closed by this PR; they should be closed as their own repositories are retrofitted.


drafted by Claude on behalf of Daniel Stephenson

dmccoystephenson and others added 2 commits August 16, 2026 10:47
…or path

Three defects observed empirically in a 10-way parallel fan-out on 2026-08-15:

- A generically-named scratchpad file holding one repo's PR body was silently
  overwritten by a concurrent dev loop. Phase 3 now requires scratch filenames
  unique per repo and per cycle, Phase 4 composes the PR body by file, and the
  concurrent-session edge case names the scratchpad as a third shared resource.
- A repository whose required jobs have never succeeded on the base branch had
  no defined disposition, so every cycle terminated in an ordinary hand-off.
  Phase 1 now establishes whether a merge path exists, Phase 4 permits a
  per-job signal assessment, and Phase 8 names the blocking condition.
- A build tool reporting no test sources still exits successfully. Phase 3 now
  requires confirming the anchor executed tests before recording a PASS.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@dmccoystephenson
dmccoystephenson merged commit 44756ec into main Aug 16, 2026
1 check passed
dmccoystephenson added a commit that referenced this pull request Aug 29, 2026
The Docs rubric item was an anchor-shaped score with no anchor behind it,
because the check it named ran three phases later. That is the same defect as
finding 1's PR #121 entry, relocated from the signal to the schedule.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dmccoystephenson added a commit that referenced this pull request Aug 29, 2026
…138)

* Ground Phase 4's Docs score in a real pass and fix two Phase 3 rules

Phase 4's Docs rubric item named a check that Phase 7 does not perform until
three phases later, so the item could only ever be scored from judgment — the
thing the rubric's own framing forbids — and a clean Phase 7 sweep left no
trace that would distinguish it from a skipped one. The table pass now happens
at Phase 4, where the score is taken; Phase 7 becomes a re-check scoped to
whatever Phases 5-6 changed, with an explicit instruction to record that it
ran.

Phase 3's zero-tests rule ended by describing {{TEST_CMD}} as a substitution,
which is the one context a generated child skill never has: after substitution
it read "fall back to whatever the ./gradlew test substitution names". It now
names the executed-test count as the gate, which reads correctly on both sides
of substitution. A scan of the rest of the template body found no other
sentence referring to a token as a token.

Phase 3's scratch-file rule gains a third cleanup form observed to work where
a bare rm is blocked: git clean -f on a single untracked path inside the
working tree, previewed with git clean -n.

Closes #126
Closes #134
Closes #137

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Record PR #138 under RESEARCH.md finding 1

The Docs rubric item was an anchor-shaped score with no anchor behind it,
because the check it named ran three phases later. That is the same defect as
finding 1's PR #121 entry, relocated from the signal to the schedule.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant